
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 13-1450 
www.uspio.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/927,041 



08/09/2001 



7590 



06/05/2006 

Thomas H. Close 
Patent Legal Staff 
Eastman Kodak Company 
343 State Street 
Rochester, NY 14650-2201 



Alexander C. Loui 



82600DMW 



4725 



EXAMINER 



HANNE, SARA M 



ART UNIT 



PAPER NUMBER 



2179 

DATE MAILED: 06/05/2006 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 


Application No. 

09/927,041 


Applicant(s) 

LOUl ETAL 


Examiner 

Sara M. Hanne 


Art Unit 

2179 





~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address -- 
Period for Reply 



A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) OR THIRTY (30) DAYS, 
WHICHEVER IS LONGER, FROM THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1)S Responsive to communication(s) filed on 06 March 2006 . 
2a)S This action is FINAL. 2b)n This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) [3 Claim(s) 1 and 3-29 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) n Claim(s) is/are allowed. 

6) ^ Claim(s) 1 and 3-29 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10) 0 The drawing(s) filed on is/are: a)n accepted or b)n objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet{s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) 0 The oath or declaration is objected to by the Examiner. Note the attached Office Action orfomn PTO-152. 

Priority under 35 U.S.C. § 119 

12) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (0- 
a)n All b)n Some * c)^ None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) D Notice of References Cited (PTO-892) 4) □ Inten/iew Summary (PTO-413) 

2) □ Notice of Drafts person's Patent Drawing Review (PTO-948) Paper No(s)/MaiI Date. . 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5) □ Notice of Informal Patent Application (PTO-1 52) 

Paper No(s)/Mail Date . 6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326(Rev. 7-05) 



Office Action Summary 



Part of Paper No./Mail Date 20060525 



Application/Control Number: 09/927,041 Page 2 

Art Unit: 2179 

DETAILED ACTION 

1 . This action is responsive to the amendment received on September 5, 2005. 
Claims 1, 3-23 are pending in the application. 



Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C, 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 



3. Claims 1, 3-8, 10 and 23-29 are rejected under 35 U.S.C. 103(a) as being 
unpatentable Qian et al., US Patent 6721454 and further in view of Ratakonda, US 
Patent 5956026. 
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As in Claims 1, 24, and 27-29, Qian et al. teaches a method and computer 
storage medium with instructions for obtaining unstructured video frames ("A video 
sequence 2 is input", Column 2, lines 64-65), generating segments from the shot 
boundaries based on the color dissimilarity between consecutive frames ("A color 
histogram technique may be used to detect the boundaries of the shots". Column 3, 
lines 42-43), extracting a set by processing pairs of segments ("the global motion of the 
video content is estimated 8 for each pair of frames in a shot", Column 3, lines 59-61) 
for their visual dissimilarity and temporal relationship, generating a feature including 
metrics of temporal separation between segments of the respective pair and 
accumulated duration of sements of the pair (temporal and spatial phenomena), and 
merging the video segments by applying a probabilistic analysis to the extracted set to 
represent the video structure ("each shot is summarized 16 ... events 22 are inferred 
from the shot summaries by a domain specific event inference model". Column 3, lines 
6-8). While Qian et al, teaches extracting semantic events from unstructured video 
frames, they fail to show the generation of inter-segment color dissimilarity feature and 
inter-segment temporal relationship feature of each pair of segments as recited in the 
claims. In the same field of the invention, Ratakonda teaches a video event detection 
and segmentation merging method similar to that of Qian et al. In addition, Ratakonda 
further teaches the generation of inter-segment color dissimilarity feature and inter- 
segment temporal relationship feature of each pair of segments (Figures 1 , 5 and 
corresponding text). It would have been obvious to one of ordinary skill in the art, 
having the teachings of Qian et al. and Ratakonda before him at the time the invention 
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was made, to modify the segment generation and merging techniques taught by Qian et 
al. to include the processing of each pair of segments of Ratakonda, in order to obtain 
not only frames, but also inter-segment similarity processing. One would have been 
motivated to make such a combination because layered hierarchical structure would 
have been obtained, as taught by Ratakonda. 

As in Claim 3, Qian et al. teaches obtaining unstructured video frames, 
generating segments from the shot boundaries based on the color dissimilarity between 
consecutive frames, extracting a set by processing pairs of segments for their visual 
dissimilarity and temporal relationship by generating color histograms from the 
consecutive frames and from the histograms, generating a difference signal, 
thresholding of this signal based on a mean dissimilarity over several frames to produce 
a signal representative of the existence of a shot boundary (See Claim 23 rejection 
supra) and merging the video segments by applying a probabilistic analysis to the 
extracted set to represent the video structure (See Claim 1 rejection supra) and the 
difference signal to be based on a mean dissimilarity over several frames centered on 
one frame. 

As in Claim 4, Qian et al. teaches morphologically transforming the thresholded 
difference signal with a pair of structuring elements to eliminate the presence of multiple 
adjacent shot boundaries ("When the difference between the histograms of two frames 
exceeds a predefined threshold, the content of the two frames is assumed to be 
sufficiently different", Column 3, lines 45-48). 
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As in Claim 5, Qian et al. teaches computing a mean color histogram for each 
segment and a visual dissimilarity feature metric from the difference between mean 
color histograms for pairs of segments (Column 3, lines 42-50 and Figure 5). 

As in Claim 6, Qian et al. teaches processing pairs of segments for a temporal 
separation between pairs of segments and for an accumulated temporal duration 
between pairs of segments ("each shot is summarized 16 ... events 22 are inferred from 
the shot summaries by a domain specific event inference model". Column 3, lines 6-8). 

As in Claim 7, Qian et al. teaches the method of claim 1 as seen supra, and 
generating parametric mixture models (summaries created by shot summarization 16, 
Figure 1 ) to represent class-conditional densities of inter-segment features (based on 
temporal information and color analysis, See Claim 1 rejection supra) of the feature set, 
parametric mixture models being statistical models (Col. 3, lines 34-35, and CoL 4, lines 
30 et seq.) and applying the merging criterion to the parametric mixture models (event 
inference 20/detected events 22, Figure 1). 

As in Claim 8, it is notoriously well known that queues are used to implement 
hierarchical displays. The examiner takes official notice of this teaching. It would be 
obvious to one of ordinary skill in the art to combine the use of the organizing video 
segments into hierarchies with a queue implementation. 

As in Claim 10, Qian et al. teaches a method and computer storage medium with 
instructions for obtaining unstructured video frames ("A video sequence 2 is input", 
Column 2, lines 64-65), generating segments from the shot boundaries based on the 
color dissimilarity between consecutive frames ("A color histogram technique may be 
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used to detect the boundaries of the shots", Column 3, lines 42^3), extracting a set by 
processing pairs of segments ("the global motion of the video content is estimated 8 for 
each pair of frames in a shot", Column 3, lines 59-61) for their visual dissimilarity and 
temporal relationship, and merging the video segments by applying a probabilistic 
analysis to the extracted set to represent the video structure ("each shot is summarized 
16 ... events 22 are inferred from the shot summaries by a domain specific event 
inference model". Column 3, lines 6-8) wherein the merging further comprises 
generating statistical models of the feature set and applying the merging criterion to the 
statistical models (Col. 3, lines 34-35, and Col. 4, lines 30 et seq.). While Qian et al. 
teaches extracting semantic events from unstructured video frames, they fail to show 
the generation of inter-segment color dissimilarity feature and inter-segment temporal 
relationship feature of each pair of segments as recited in the claims. In the same field 
of the invention, Ratakonda teaches a video event detection and segmentation merging 
method similar to that of Qian et al. In addition, Ratakonda further teaches the 
generation of inter-segment color dissimilarity feature and inter-segment temporal 
relationship feature of each pair of segments (Figures 1 , 5 and corresponding text). It 
would have been obvious to one of ordinary skill in the art, having the teachings of Qian 
et al. and Ratakonda before him at the time the invention was made, to modify the 
segment generation and merging techniques taught by Qian et al. to include the 
processing of each pair of segments of Ratakonda, in order to obtain not only frames, 
but also inter-segment similarity processing. One would have been motivated to make 
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such a combination because layered hierarchical structure would have been obtained, 
as taught by Ratakonda. 

As in Claim 23, Qian et al. teaches generating color histograms from the 
consecutive frames and from the histograms, generating a difference signal, 
thresholding of this signal based on a mean dissimilarity over several frames to produce 
a signal representative of the existence of a shot boundary (Column 3, lines 42-50 and 
Figure 5). 

As in Claim 25, Qian et al. teaches generating parametric mixture models 
(summaries created by shot summarization 16, Figure 1) to represent class-conditional 
densities of inter-segment features (based on temporal information and color analysis, 
See Claim 1 rejection supra) of the feature set, and applying the merging criterion to the 
parametric mixture models (event inference 20/detected events 22, Figure 1 ). 

As in Claim 26, Qian et al. teaches introducing each feature to a list of shots with 
a priority equal to the probability of merging each corresponding pair of segments, 
depleting the list of segments by merging the video segments if the merging criterion is 
met, and updating the model of the merged segment and then updating the list based 
on the updated model (Figure 7 and corresponding text). It is notoriously well known 
that queues are used to implement hierarchical displays. The examiner takes official 
notice of this teaching. It would be obvious to one of ordinary skill in the art to combine 
the use of the organizing video segments into hierarchies with a queue implementation. 
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4. Claims 9, and 1 1-22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Qian et al., US Patent 6721454 and Ratakonda, US Patent 5956026 and further in 
view of Qian et a!., US Patent 6616529. 

As in Claims 9, 11, 17-18 and 20, US Patent 6721454 and Ratakonda teach a 
method and computer storage medium with instructions for obtaining unstructured video 
frames, generating segments from the shot boundaries based on the color dissimilarity 
between consecutive frames, extracting a set by processing pairs of segments for their 
color dissimilarity and temporal relationship of each pair of segments, merging adjacent 
video segments by applying a probabilistic analysis to the extracted set to represent the 
video structure, and generating a parametric mixture model of the inter-segment 
features (See Claim 1 rejection supra), the parametric mixture models being a statistical 
model (Col. 3, lines 34-35, and Col. 4, lines 30 et seq.). While US Patent 6721454 and 
Ratakonda teach the segmentation due to color dissimilarity, extraction due to visual 
dissimilarity and temporal relationships, merging with probabilistic analysis and 
generation of a parametric mixture model, they fail to show the probabilistic analysis to 
be a Bayesian analysis applied to the parametric mixture model, and representing the 
merging sequence in a hierarchical tree structure as recited in the claims. US Patent 
6616529 teaches a video segmentation method similar to that of US Patent 6721454 
and Ratakonda. In addition, US Patent 6616529 further teaches the probabilistic 
analysis to be a Bayesian analysis applied to the parametric mixture model (Figure 3 
and corresponding text in Columns 4-5), and representing the merging sequence in a 
hierarchical tree structure (Figures 2a-2g and corresponding text). It would have been 
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obvious to one of ordinary skill in the art, having the teachings of US Patent 6721454 
and Ratakonda and US Patent 6616529 before him at the time the invention was made, 
to modify the segmentation with color dissimilarity and temporal relationships with a 
parametric mixture model taught by US Patent 6721454 and Ratakonda to include the 
construction of hierarchy according to probabilistic merging with Bayesian analysis of 
US Patent 6616529, in order to obtain a hierarchical representation of the frames 
grouped by color dissimilarity and temporal relationships according to Bayesian 
probability methods of analysis. One would have been motivated to make such a 
combination because a visual representation of the segmented video would have been 
obtained, as taught by US Patent 6616529 (Column 2, lines 24-55). 

As in Claim 12, US Patent 6721454 and Ratakonda teach computing a mean 
color histogram for each segment and a visual dissimilarity feature metric from the 
difference between mean color histograms for pairs of segments (See Claim 5 rejection 
supra). 

As in Claim 13, US Patent 6721454 and Ratakonda teach processing pairs of 
segments for a temporal separation between pairs of segments and for an accumulated 
temporal duration of pairs of segments (See Claim 6 rejection supra). 

As in Claim 14, US Patent 6721454 and Ratakonda teach generating parametric 
mixture models to represent class-conditional densities of the inter-segment features 
that comprise the feature set (See Claim 7 rejection supra). 

As in Claim 15, US Patent 6721454 and Ratakonda teach performing the 
merging in a hierarchical queue by initializing the queue by introducing each feature in 
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the queue with a priority of the probability of merging each corresponding pair of 
segments, depleting the queue by merging the segments if the criterion is met, and 
updating the queue based on the updated model (See Claim 8 rejection supra). 

As in Claim 16, US Patent 6721454 and Ratakonda teach representing the 
merging sequence as a hierarchical tree structure (See Claim 9 rejection supra) 
including a frame extracted from each segment and displayed at each node of the tree 
(Column 1 0, line 61 - Column 1 1 , line 6). 

As in Claim 19, US Patent 6721454 and Ratakonda teach representing the 
merging sequence as a hierarchical tree structure including a frame extracted from each 
segment and displayed at each node of the tree (See Claim 16 rejection supra). 

As in Claim 21 , US Patent 6721454 and Ratakonda teach a method and for 
generating video segments from the unstructured video frames ("A video sequence 2 is 
input". Column 2, lines 64-65), by detecting shot boundaries based on the color 
dissimilarity between consecutive frames ("A color histogram technique may be used to 
detect the boundaries of the shots". Column 3, lines 42-43), extracting a feature set by 
processing pairs of segments ("the global motion of the video content is estimated 8 for 
each pair of frames in a shot". Column 3, lines 59-61) for their visual dissimilarity and 
temporal relationship, generating a feature including metrics of temporal separation 
between segments of the respective pair and accumulated duration of sements of the 
pair (temporal and spatial phenomena), and merging adjacent video segments by 
applying a probabilistic analysis to the feature set to represent the video structure, the 
merging being independent of any empirical parameter determination ("each shot is 
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summarized 16 ... events 22 are inferred from the shot summaries by a domain specific 
event inference model". Column 3, lines 6-8). While US Patent 6721454 teaches the 
segmentation due to color dissimilarity, extraction due to visual dissimilarity and 
temporal relationships, merging with probabilistic analysis and generation of a 
parametric mixture model, they fail to show generating a hierarchy having a merging 
sequence represented by a binary partition tree as recited in the claims. US Patent 
6616529 teaches a video segmentation method similar to that of US Patent 6721454. 
In addition, US Patent 6616529 further teaches generating a hierarchy having a 
merging sequence represented by a binary partition tree (Figures 2a-2g and 
corresponding text). It would have been obvious to one of ordinary skill in the art, 
having the teachings of US Patent 6721454 and US Patent 6616529 before him at the 
time the invention was made, to modify the segmentation with color dissimilarity and 
temporal relationships with a parametric mixture model taught by US Patent 6721454 to 
include the construction of hierarchy having a merging sequence represented by a 
binary partition tree of US Patent 6616529, in order to obtain a hierarchical 
representation of the frames grouped by color dissimilarity and temporal relationships. 
One would have been motivated to make such a combination because an organized 
visual representation of the segmented video would have been obtained, as taught by 
US Patent 6616529 (Column 2, lines 24-55). 

As in Claim 22, US Patent 6616529 teaches maximizing the a posteriori 
probability mass function of a binary random variable that represents inter-segment 
features of the video segments (Figures 2a-2g and Column 2, lines 45, et seq.j. 
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Response to Arguments 

Applicant's arguments filed 3/6/06 have been fully considered but they are not 
persuasive. In response to the arguments regarding the claim new amendments, see 
rejections supra. 

In response to the arguments regarding motivation to combine Qian and 
Ratakonda the examiner disagrees. Ratakonda clearly teaches layers of a parent/child 
hierarchy structure and describes the advanatages of such structure (Col. 1 , line 64 et 
seq.). 

In response to the arguments stating that Qian fails to teach descriptor of 
different shots are compared but not in pairs, the examiner disagrees. Qian teaches 
each pair of frames compared (Col. 3, lines 59 et seq.) 

In response to the arguments regarding claim 4, Qian does teach eliminating the 
presence of multiple adjacent shot boundaries. Even if they do not, this can be seen 
also by Ratakonda in the higher levels of the hierarchy. 

In response to the arguments regarding claim 5 and 6, Ratakonda teaches 
processing each pair of segments for dissimilarity in the same way Qian does for frames 
as seen supra. 

In response to the arguments regarding claim 8 and 15, Qian teaches the 
process of "inserting" merges frames together, constituting a pair of segments that 
define the event and updating the model of the merged segment. Ratakonda further 
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illustrates step d as seen supra. Furthermore, claim 15 is interepretted with respect to 
the official notice of Claim 8. Claim 15 merging and depleting sequence can further be 
illustrated by Qian figures 1 and 7 with corresponding text. 
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Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

The prior art made of record on form PTO-892 and not relied upon is considered 
pertinent to applicant's disclosure. Applicant is required under 37 C. F. R. § 1 . 1 1 1 (c) to 
consider these references fully when responding to this action. The documents cited 
therein teach similar video segment merging techniques. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Sara M. Hanne whose telephone number is (571 ) 272- 
4135. The examiner can normally be reached on M-F 7:30am-4:00pm, off on 
alternating Fridays. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, WEILUN LO can be reached on (571) 272-4847. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (BBC) at 866-217-9197 (toll-free). 

smh 
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